Using the EWM REST and OSLC APIs

I have collected a lot of experience with the Java APIs for RTC/EWM over the years. Until 2020 I did not use the RTC/EWM REST and OSLC APIs at all. Luckily I got involved in several engagements where I had the opportunity to explore these APIs and learn how to use them.

I found the documentation for these APIs I was able to find underwhelming. The available documentation was often lacking complete working examples. There was usually some critical part missing or there where no examples at all, just the API specification. The latter seemed to be systematic to focus on the specification and not the specifics. However the lack of samples was confusing and left too much room for interpretation. I ended using search engines a lot. I had to experiment a lot to get things moving.

Obviously I learned a lot. I would like to spare others the hassle. So, as usual I want to share the experiences and lessons learned with the community. My intention is to provide with some relevant and working examples that are easy enough to perform on your own. I hope this can save people some time when trying to use these APIS as well.

This post will be the first in a series of posts and provide links to the other posts for easy navigation. In addition this post will discuss which development environment and tools were utilized to explore and use the APIs. I will share some of the code I have developed over the time to ease the exploration.

The planned blog posts for the series

This is the series of planned posts I intent to publish over time. Most of the examples will be EWM based, but quite a lot of the content applies to more ELM applications. The examples where performed with versions 6.0.6.1 and 7.0.x.

• Using the EWM REST and OSLC APIs – this post
• ELM Authentication
• EWM Discovery
• EWM Work Item OSLC CM API
• EWM OSLC Query API
• EWM REST API to access existing Work Item Queries
• EWM Reportable REST API

Important links about the EWM/RTC APIs and the ELM APIs in general

I wrote Learning To Fly: Getting Started with the RTC Java API’s a couple of years back and it is still relevant. Read it to understand What API’s are Available for RTC and What Can You Extend? and how to get started with the RTC/EWM Java APIs. The post and the linked posts contain even more valuable links with respect to APIs.

Since back then, the ELM API Landing Page has been added to provide a more comprehensive overview about the available ELM APIs. If you are interested in ELM related APIs go over that page and find out what APIs are available. This page also points to other resources such as the OSLC standards and available workshops.

Finally the Interesting links page is a collection of, well, interesting links I found over the years.

Development environment choices

The first information I can share is how I explored and used the APIs and explain a little bit about the development environment options available and which development environments I used to explore the APIs.

I like developing with Java a lot. The EWM/RTC Java APIs are very rich and it is relatively easy to develop code for EWM/RTC, provided the development environment was set up by performing at least the complete Lab 1 of the Rational Team Concert Extensions Workshop. Eclipse, the RTC SDK and the Plain Java Client Libraries allow development of extensions and automation based on the EWM Java SDK and API. The same environment can also be used to develop code against the REST and OSLC APIs.

It is also possible to use Java or any other language supporting HTTP, to develop code for the EWM/RTC REST and OSLC APIs, just using libraries and available frameworks.

I have already used the Java based Eclipse Lyo framework to develop a client automation for Doors Next Generation (DNG) and I used the Eclipse Lyo Designer code generation framework to develop integration servers. My experience was that Lyo is a nice framework that helps a lot, if you know what you are doing. If I was not, I found it challenging, especially debugging and understanding what was going on in the HTTP requests.

I have looked into and used Postman and the Firefox addon RESTClient to experiment with REST and OSLC APIs. It is very useful for experimentation and I use it in parallel to the other development environments. A typical use case is to login and experiment with one call to figure out how it works. If the call sequence and the amount of data becomes too big, it is not really efficient any more, and I would use a different approach.

I started using Python and Jupyter Notebook in 2020 when I had the need for some automation for importing, manipulating, consolidating and querying a lot of CSV data for a customer. I was very impressed with the quality of the available libraries and the turn around times that were achievable. When I was asked to help one of our customer teams with information about the RTC/EWM APIs for the development of a prototype for a customer specific mobile client, I decided to use Python instead of Java. As mentioned above I also used RESTClient or Postman for experimentation with one or two API requests.

There are various Python development environments around. I do not think it matters which one you use. I used Spyder which comes with Anaconda. There is also PyCharm and kite. I am not opinionated. I just notice that the development environments are far away from the quality of Eclipse and the built in compiler and debugger. There are always tradeoffs, I guess.

Python – Libraries and Code Samples

The focus of the blog posts is more the APIs, how they work and how they can be used and not so much Python and how to use it. However, I figured that I want to share some code I developed over time, enabling easier data collection and debugging. So I will provide examples where I see fit.

The most important aspects of HTTP based APIs is to understand which method is used with which URI, which headers are used, which formats are sent and accepted and which request body (if any) is sent. The response data is also key, especially the status, response headers such as Location and obviously the response body. A mechanism that can log all this information is key in understanding the APIs and faster turn around times.

Python has a lot of libraries for various purposes. The Libraries that I used are shown below, loosely grouped by what they are used for. First the libraries used operating system and system specific purposes such as logging, files and execution.

import os
import sys
from datetime import datetime, timezone
from pathlib import Path

Then the requests library which is used for session handling and HTTP communication in my code.

import requests
from requests.auth import HTTPBasicAuth
from requests.packages.urllib3.exceptions import InsecureRequestWarning

This code below is necessary to suppress issues with certificates. This is a typical situation for me as I usually develop against some local test system.

# Disable warnings for self signed or invalid SSL certificates 
# to be able to talk to test systems
requests.packages.urllib3.disable_warnings(InsecureRequestWarning)

# Start a session
session = requests.Session()

The Libraries I used for RDF XML and JSON parsing and representation.

import json

from rdflib import Graph, URIRef, Namespace
from rdflib.namespace import CSVW, DC, DCAT, DCTERMS, DOAP, FOAF, ODRL2, ORG, OWL, PROF, PROV, RDF, RDFS, SDO, SH, SKOS, SOSA, SSN, TIME, VOID, XMLNS, XSD

Some miscellaneous library for encoding.

from base64 import b64encode

Python Logging and Reuse

I ended up creating a base library for the Communication with the ELM system that allowed better reuse. I will not share all the code at the moment, but I will share some basic learning and code that I found being key for me to be able to do my work. The library is referred to as:

from elmcommlib import ELMCommLib as elmcomm

The library is initialized with a session, the public URI and a name for the log folder to be created.

publicURI = 'https://elm.example.com:9443/ccm'
paName='JKE Banking (Change Management)'
user='ralph'
password='ralph'

# Disable warnings for self signed or invalid SSL certificates 
# to be able to talk to test systems
requests.packages.urllib3.disable_warnings(InsecureRequestWarning)

# Start a session
session = requests.Session()
comm=elmcomm(session, publicURI,'logCreateWiRDF')

The library also provides a mechanism to create and set log file folders using createLogFolder("FolderName") . In case the folder already exists it can alternatively set with setLogFolder("FolderName").

The log folder is used by the method writeResult() shown below, which dumps the complete communication in a text file, when a file name is provided. The file name should be constructed and numbered to better understand the flow of the sequences. Below shows such a sequence with file name numbering as an example.

The communication logs are always created in the current log folder. This allows to split the logs for the API usage into smaller sequences by switching the current log folder.

Content of a log folder.

A debug print dPrint() allows to avoid chatty logging. You can keep the logging entry and force it to show if you want. Printing a timestamp using timeStamp() is sometimes useful, especially when looking at performance of calls.

    # Folder for log files
    def createlogFolder(self,folderName):
        defaultLogFolder= 'commlogs'
        if(folderName==None):
            folderName=defaultLogFolder
        if(folderName==''):    
            folderName=defaultLogFolder
        script_dir = os.path.dirname(__file__) #<-- absolute dir the script is in
        logFolder=os.path.join(script_dir, folderName)
        Path(logFolder).mkdir(parents=True, exist_ok=True)
        return logFolder

    # Folder for log files
    def setlogFolder(self, folderName):
       self.logFolder=self.createlogFolder(folderName)


    # Log the HTTP communication for the request in a folder
    def writeResult(self, fileName, result, url=None):
        if(fileName!='fileName'):
            self.dPrint(f"Execute: '{fileName}'")
            logFileName=os.path.join(self.logFolder, fileName)
            with open(logFileName,'w') as f:
                if(url!=None):
                    f.write(f"Destination URL: {url}\n\n")
                reqMethod = result.request.method
                reqURL=result.request.url
                reqBody=result.request.body
                f.write(f"Request: {reqMethod} {reqURL}\n")
                f.write("\nRequest Headers:\n")    
                for header in result.request.headers:
                    value=result.request.headers[header]
                    f.write(f"\t{header} {value}\n")
                f.write(f"\nBody:\n{reqBody}\n") 
                f.write(f"\nResponse Status: {result.status_code}\n")
                f.write("\nResponse Headers:\n")
                for header in result.headers:
                    value=result.headers[header]
                    f.write(f"\t{header} {value}\n")
                cookies=result.cookies._cookies    
                f.write(f"\nResponse Cookies:\n {cookies}\n")
                f.write(f"\nResult Body:\n{result.text}\n")        
      
      
    # Debug print if debugging is on
    def dPrint(self, message=None, doPrint=True):
        # DebugPrint, switch off by sending doPrint=False
        if(message!=None):
            if(doPrint==True):
                print(message)
            else:
                pass
    

    # Print timestamp
    def timeStamp(self, message):
        now = datetime.now(timezone.utc)
        self.dPrint(f"{now}: {message}")
   

The image below shows how a log file for a message created using the method writeResult() looks like. Note that the log contains all the important pieces of the request, response pair. I used tooling in the editor Notepad++ to “pretty print” format the XML section in the response body. This makes it much easier to understand.

Logged http request – response

Help with RDF XML

The REST and OSLC APIs provide different serializations for the content that they accept and provide. One older one is XML based using the Resource Description Framework (RDF) specification. Newer standards such as JSON and more might be available. I have experience with RDF and JSON and I prefer JSON.

RDF is not for me. I always struggle to understand what and how I should be searching to get the data I want. Especially when the data is full blown with namespaces and what have you. This was one of the biggest struggles I had with Eclipse Lyo. The HTTP Client was hard to use for debugging, because the content was usually consumed when I tried to dump the response into a log file. So I could have a log entry for debugging and the call would not proceed or the call would work and I had no log data. Maybe I overlooked something.

In Python I was able to use the method writeResult() and continue processing the response data. I was able to use the function below to serialize RDF response bodies into a form that shows all the subject, predicate and object data and saves it into a file. That made it easier to work with RDF for me. I still prefer JSON format, if available. The OSLC discovery mechanism supported by RTC/EWM requires XML-RDF in the first steps, so you will have to deal with it.

    # Serializes a graph (based on RDF) in the nt format 
    # This format shows all graph nodes as Subject->Predicate->Object
    # This allows to better understand what to search for
    def debugSerialize(self, graph, fileName='fileName'):
        # Serializes a graph into the NT format. This provides 
        # a great source to look into RDF triples in the graph
        if(fileName!='fileName'):
            logFileName=os.path.join(self.logFolder, fileName)
            graph.serialize(logFileName, format="nt")

The serialization formats supported out of the box are “xml”, “n3”, “turtle”, “nt”, “pretty-xml”, “trix”, “trig” and “nquads”. For me NT and Turtle seem to be most useful, I built in capabilities to save the XML data as NT and Turtle format to help understanding how to be able to access the data later.

Update: The preferred option is to serialize the graph as Turtle format and look into that.

This is how the parsed RDF graph data from above looks in the NT format. Every row (mind the word wrap) is a triple of subject, predicate and object. This provides with hints how to search for data.

The RDF-XML in NT format, providing the triples in the model.

The capabilities above where absolute key for me to be able to explore and understand the EWM/RTC APIs and document them for my colleagues.

Operating on RDF requires the RDF definitions in Python. I used the ones below and defined them in my library.

    #RTC CM RDF definitions
    rtc_cm = 'rtc_cm'
    rtc_cm_URI = 'http://jazz.net/xmlns/prod/jazz/rtc/cm/1.0/'
    rtc_cm_ns = Namespace(rtc_cm_URI)
    
    oslc = 'oslc'
    oslc_URI = 'http://open-services.net/ns/core#'
    oslc_ns = Namespace(oslc_URI)
    
    oslc_cm = 'oslc_cm'
    oslc_cm_URI = 'http://open-services.net/ns/cm#'
    oslc_cm_ns = Namespace(oslc_cm_URI)
    
    oslc_xml_cm1 = 'oslc_cm1'
    oslc_xml_cm1_URI = 'http://open-services.net/xmlns/cm/1.0/'
    oslc_xml_cm1_ns = Namespace(oslc_xml_cm1_URI)
    
    jfs_process = 'jfs_proc'
    jfs_process_URI ='http://jazz.net/xmlns/prod/jazz/process/1.0/'
    jfs_process_ns = Namespace(jfs_process_URI)

    oslc_rm = 'oslc_rm'
    oslc_rm_URI = 'http://open-services.net/xmlns/rm/1.0/'
    oslc_rm_ns = Namespace(oslc_rm_URI)
    
    oslc_config = 'oslc_config'
    oslc_config_URI = 'http://open-services.net/ns/config#'
    oslc_config_ns = Namespace(oslc_config_URI)

Summary

This is the first of a series of posts, I hope to publish more soon. I will try to keep this post maintained and I am looking forward to the next posts. As always, I hope that my content, especially in this blog, helps someone in the ELM community out there. If it does, feedback would be awesome.